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Top  Five  Systems  Engineering  Issues* 


•  Lack  of  awareness  of  the  importance,  value,  timing, 
accountability,  and  organizational  structure  of  SE  on 
programs 

•  Adequate,  qualified  resources  are  generally  not  available 
within  government  and  industry  for  allocation  on  major 
programs 

•  Insufficient  SE  tools  and  environments  to  effectively  execute 
SE  on  programs 

•  Requirements  definition,  development,  and  management  is 
not  applied  consistently  and  effectively 

•  Poor  initial  program  formulation 


*  Based  on  an  NDIA  Study  in  January  2003  2 


USD(AT&L)  Imperative 


I  should  note ...  that  we  have  taken  important  steps  that 
will  help  us  to  produce  improved  capability  on  time  and 
within  budget  by  re-energizing  our  approach  to  systems 
engineering.  This  critical  discipline  has  always 
contributed  significantly  to  effective  program 
management  at  every  level  and  will  receive  sustained 

emphasis  during  my  tenure. 

Testimony  of  The  Honorable  Kenneth  J.  Krieg,  USD(AT&L), 
before  US  Committee  on  Armed  Services,  September  27,  2005 
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What  We  Have  Done  To  Revitalize 
Systems  Engineering 


•  Established  SE  Forum — senior-level  focus  within  DoD 

•  Issued  Department-wide  systems  engineering  (SE)  policy 

•  Issued  guidance  on  SE  and  test  and  evaluation  (T&E) — 
focused  on  effective,  early  engagement  of  both 

•  Instituted  system-level  assessments  in  support  of  OSD  major 
acquisition  program  oversight  role 

•  Working  with  Defense  Acquisition  University  to  revise  SE, 
T&E,  and  enabling  career  fields  curricula  (Acq,  PM,  CM,  FM) 

•  Leveraging  close  working  relationships  with  industry  and 
academia 


http://www.acq.osd.mil/ds/se/index.html 


DoD  Response 
Policy  and  Guidance 


•  Policy 

-  Programs  shall  apply  robust  SE  approach  and  develop  a  SE  Plan 

-  Each  PEO  shall  have  a  lead  or  chief  systems  engineer 

-  Event-driven  technical  reviews  with  entry  criteria  and  independent 
SMEs  unless  waived  by  MDA 

-  OSD  shall  review  program  SEPs  for  ACAT  ID  and  1AM  programs 

-  Defense  Systems  shall  establish  a  SE  Forum 

•  Guidance 

-  Defense  Acquisition  Guidebook :  SE  in  DoD  Acquisition,  SE 
Processes,  SE  Implementation  in  the  System  Life  Cycle,  SE  Tools 
and  Techniques,  and  SE  Resources;  Test  &  Evaluation 

-  Systems  Engineering  Plan:  interim  guidance,  Preparation  Guide,  and 
five  focus  areas  to  address  in  technical  planning  throughout  the 
system  life  cycle 


Striving  for  Technical  Excellence 


•  All  programs  shall  develop  a  SE  > 
Plan  (SEP) 

•  Each  PEO  shall  have  a  lead  or 
chief  systems  engineer  who 
monitors  SE  implementation 
within  program  portfolio 

•  Event-driven  technical  reviews 
with  entry  criteria  and  independent 
subject  matter  expert  participation 

•  OSD  shall  review  program’s  SEP 
for  major  acquisition  programs 
(ACAT  ID  and  I  AM) 
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Strong  technical  foundation  is  the  value  of 
SE  to  the  program  manager 
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Why  Plan? 


Technical  Planning  Drivers 
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Benefits  of  Technical  Planning 


•  Technical  planning  fosters  escalating  - ►  •  An  executable  Acquisition  Strategy 

understanding  of  the  technical  challenge 

and  the  insightful  data  for  establishing 
executability 

•  Technical  planning  fosters  escalating  - ►  •  A  valid  Cost  Estimate 

understanding  of  the  technical  challenge, 

providing  increased  confidence  over  time 
in  a  valid  cost  estimate 

•  SE  process  critical  to  requirements  - ►  •  A  valid  Requirement 

maturation  from  concept  through  test 

•  Technical  planning  ensures  verification  - ►  •  A  report  from  the  Evaluator 

and  validation  are  part  of  the  technical 

baseline 

•  Technical  planning  integrates  sustaining  - ►  •  A  plan  for  Sustainment 

engineering  as  a  readiness  enabler 

What  Decision  Makers  Need  To  Make  An 

Informed  Decision 
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Why  Document  the  Plan? 


SEP  Stakeholders 


Other  Programs 

Statutory  and  Regulatory 
Bodies 

Certifiers 
Users 

Cost  Estimators 
New  Program  Personnel 


Program  Manager 

PEO  Lead  Systems  Engineer 


Prime  Contractor 
Subcontractors 
Lower-tier  Suppliers 
Functional  Leadership 
IPTs 
Testers 
Logisticians 


Milestone  Decision  Authority 


A  SEP  Provides  a  Means  for  Collective 
Understanding  Among  All  Stakeholders  as  to 
Program’s  Technical  Approach 
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Driving  Technical  Rigor  Back  into  Programs 
“Importance  /  Criticality  of  Technical  Planning” 


•  Program’s  SEP  provides  insight  into  every  aspect  of  a 
program’s  technical  planning,  focusing  on: 

-  What  are  all  the  program  requirements? 

-  Who  has  responsibility  and  authority  for  managing  technical  issues — 
what  is  the  staffing  and  organization  to  support  the  effort? 

-  How  will  the  technical  baseline  be  managed  and  controlled? 

-  What  is  the  technical  review  process? 

-  How  is  that  technical  effort  linked  to  overall  management  of  program? 

•  Living  document  with  use,  application,  and  updates  clearly 
evident 

The  SEP  is  fundamental  to  technical  and 
programmatic  execution  on  a  program 
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SEP  Observations 


•  Descriptions  vice  plans 

-  Regurgitated  theory 

-  Generic  text,  applicable  to _ 

-  Disconnected  discussion 

-  No  numbers  or  specifics 

-  No  names 

-  No  timeframes  or  ordered  relationships 

•  Not  reflective  of  known  industry  best  practice 

-  Technical  baselines 

-  Technical  reviews 

•  Entry  criteria  for  technical  reviews 

•  Peer  participation 


What 

How 

Why 

Where 

Who 

When 


13 


Driving  Technical  Rigor  Back  Into  Programs 
“Emerging  SEP  Comments  (First  Drafts)” 

(not  systemic  across  all  programs) 


•  Incomplete  discussion  of  program  requirements 

-  Missing  categories  such  as  statutory,  regulatory,  or  certifications 

•  Minimal  discussion  of  program  IPTs 

-  Need  to  identify  technical  authority,  lead  systems  engineer,  and  key  stakeholders 

-  Addresses  part  of  SE  organization,  such  as  prime;  no  mention  of  government,  subcontractors,  or 
suppliers 

•  Incomplete  technical  baseline 

-  How  does  the  program  go  from  CDD  to  product — traceability? 

-  Linkage  to  EVM — not  able  to  measure  technical  maturity  via  baselines 

•  Incomplete  discussion  of  technical  reviews 

-  How  many,  for  what  (should  tie  to  baselines  and  systems/subsystems/configuration  items),  and  by 
whom  (should  tie  to  staffing)? 

-  Lacking  specific  entry  criteria 

-  Peer  reviews 

•  Integration  with  other  management  planning 

-  Linkage  with  acquisition  strategy,  IMP,  IMS,  logistics,  testing, 
and  risk  management 

-  Schedule  adequacy — success-oriented  vice  event-driven; 
schedule  realism 

-  Contracting  for  SE 


Compelling  Need  to  Engage  with  Programs  Early  in  Process 


75  SEPs 
reviewed 
from  46 
programs 
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When  Should  Technical 
Planning  Occur? 


Who  Should  do  It? 


Scope  of  Technical  Planning 


Technical  Reviews  Interactive  Timeline 


Ph  ase 


Work 

Efforts^ 


Activities 

Technical 


Sound  technical  planning  is  needed  in  EVERY  acquisition  phase 
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Technical  Planning  Timeline 


•  RFP  Preparation 

•  Acquirer’s  Technical 
Approach  as 
Documented  in 
Draft  SEP 

•  Written  by  Program 
Manager,  Lead  SE, 
Lead  Tester,  and 
Lead  Logistician 


Milestone 


•  Source  Selection 

•  Offeror’s 
Proposed 
Technical 
Approach  based 
on  Draft  SEP 

•  Evaluated  by 
Source  Selection 
Evaluation  Board 


•  Post-Award  Planning 

•  Program  Team’s 
Technical  Approach  as 
Documented  in  Program 
SEP 

•  Written  by  Program 
Manager,  Lead  SE, 

Lead  Tester,  and  Lead 
Logistician  from 
Government,  Prime, 
Subs,  and  Suppliers 


Execution 

•  Execute  the 
Technical 
Approach 

•  Updated  by 
Program 
Team 


A  shared  “vision”  of  SE  on  your  program 
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What  should  be  addressed 
in  a  sound  technical  plan 
for  a  program? 


Technical  Planning  Considerations 


Program  Acquisition 
Objectives 

User  Need 


Technology 

Maturity 

Budget 

Limitations 


Service  /  Agency 
Enterprise 
Considerations 


Defense  Acquisition 
Guidebook 


Technical 

Planning 


OSD  SEP 
Preparation  Guide 


Service  /  Agency 
Unique  Guidance 


This  is  the  Program  Manager’s  Planning! 
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Guidance  and  Tools 


Defense  Acquisition  Guidebook : 

-  SE  in  DoD  Acquisition 

-  SE  Processes 

-  SE  Implementation  in  the  System  Life  Cycle 

-  SE  Tools  and  Techniques,  and  SE  Resources 

-  Life  Cycle  Logistics  in  SE  - 

-  Test  &  Evaluation  - 

SEP: 

-  Interim  guidance 

-  Preparation  Guide 

-  Twenty-five  focus  areas  to  address  in  technical  planning 


WChapter  4 


Chapter  5 
■♦\Chapter  9, 
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SE  in  the  System  Life  Cycle 
“The  Wall  Chart” 
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SE  in  the  System  Life  Cycle 
Defense  Acquisition  Guidebook 
Chapter  4,  Section  4.3 


•  By  phase  consideration  of  SE  activities 

-  Purpose  of  SE  in  the  phase 

-  Inputs  to  the  SE  process 
-Key  SE  activities  during  the  phase 
-Technical  reviews  during  the  phase 
-Outputs  of  the  phase’s  SE  process 

•  Full  life  cycle  coverage  from  Concept  Refinement 
through  Operations  and  Support 
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Guidance  and  Tools 


Defense  Acquisition  Guidebook : 

-  SE  in  DoD  Acquisition 

-  SE  Processes 

-  SE  Implementation  in  the  System  Life  Cycle 

-  SE  Tools  and  Techniques,  and  SE  Resources 

-  Life  Cycle  Logistics  in  SE  - 

-  Test  &  Evaluation  - 

SEP: 

-  Interim  guidance 

-  Preparation  Guide 

-  Twenty-five  focus  areas  to  address  in  technical  planning 


WChapter  4 


Chapter  5 
+\Chapter  9. 
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Driving  Technical  Rigor  Back  Into  Programs 
SEP  Focus  Areas  for  Technical  Planning  in 
SDD/Production  and  Deployment 


•  Program  Requirements 

-  Capabilities,  CONOPS,  KPPs 

-  Statutory/regulatory 

-  Specified/derived  performance 

-  Certifications 

-  Design  considerations 

•  Technical  Staffing/Organization 

-  Technical  authority 

-  Lead  Systems  Engineer 

-  IPT  coordination 

-  IPT  organization 

-  Organizational  depth 

•  Technical  Baseline  Management 

-  Who  is  responsible 

-  Definition  of  baselines 

-  Requirements  traceability 

-  Specification  tree  and  WBS  link 

-  Technical  maturity  and  risk 


•  Technical  Review  Planning 

-  Event-driven  reviews 

-  Management  of  reviews 

-  Technical  authority  chair 

-  Key  stakeholder  participation 

-  Peer  participation 

•  Integration  with  Overall  Management 
of  the  Program 

-  Linkage  with  other  program  plans 

-  Program  manager’s  role  in  technical 
reviews 

-  Risk  management  integration 

-  Test  and  logistics  integration 

-  Contracting  considerations 


Adapt  for  Domain  and  Enterprise  Considerations 
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How  would  technical  planning 

need  to  change  to 
accommodate  programs  Pre- 
Milestone  B  and  Post- 
Milestone  C? 


Driving  Technical  Rigor  Back  Into  Programs 
SEP  Focus  Areas  for  Technical  Planning  in 
Concept  Refinement  /  Technology  Development 


Program  Requirements 

-  Desired  capabilities;  required 
attributes 

-  Potential  statutory/regulatory, 
specified/derived  performance, 
certifications,  design  considerations 

-  Enabling  technologies 

-  Cost/schedule  constraints 

-  Future  planning 

Technical  Staffing/Organization 

-  Technical  authority 

-  Lead  Systems  Engineer 

-  SE  role  in  TD  IPT 

-  IPT  organization  and  coordination 

-  Organizational  depth 


Technical  Baseline  Management 

-  Who  is  responsible 

-  Definition  of  baselines 

-  ICD/CDD  traceability 

-  Technical  maturity  and  risk 

Technical  Review  Planning 

-  Event-driven  reviews 

-  Management  of  reviews 

-  Technical  authority  chair 

-  Key  stakeholder  participation 

-  Peer  participation 

Integration  with  Overall  Management 
of  the  Program 

-  Linkage  with  other  program  plans 

-  Program  manager’s  role  in  reviews 

-  Risk  management  integration 

-  Test  and  support  strategy 

-  Contracting  considerations 


Adapt  for  Domain  and  Enterprise  Considerations 
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Driving  Technical  Rigor  Back  Into  Programs 
SEP  Focus  Areas  for  Technical  Planning  in 

Sustainment 


•  Program  Requirements 

-  Technical  surveillance  approach 

-  Tracking  of  actual  vs.  planned  usage 

-  Monitoring  of  system  hazards,  risks, 
certifications 

-  Tracking  of  usage,  corrosion-related 
maintenance  and  repair  costs,  and  total 
ownership  costs 

-  Management  of  configuration  changes 
and  incremental  modifications 

•  Technical  Staffing/Organization 

-  Technical  authority 

-  Lead  Systems  Engineer 

-  Coordination  of  sustaining  engineering 
with  operational,  maintenance,  and  repair 
domains 

-  Sustaining  support  organization 

-  Organizational  depth 


•  Technical  Baseline  Management 

-  Who  is  responsible 

-  Definition  of  baseline  management 

-  Requirements  and  certification 
traceability  and  verification  of  changes 

-  Specification  tree  and  WBS  link 

-  Tracking  of  operational  hazard  risk 
against  baseline 

•  Technical  Review  Planning 

-  In-service  reviews 

-  Management  of  reviews 

-  Technical  authority  chair 

-  Key  stakeholder  participation 

-  Peer  participation 

•  Integration  with  Program  Management 

-  Linkage  with  overall  sustainment 

-  Program  manager’s  role  in  in-service 
reviews 

-  Risk  management  integration 

-  Logistics  integration 

-  Contracting  considerations 


Adapt  for  Domain  and  Enterprise  Considerations 
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Systems  Engineering  Revitalization 

Framework 


Driving  Technicai  Excellence  into  Programs! 
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Summary 


•  OSD’s  fundamental  role  is  to  set  policy,  provide  relevant  and 
effective  education  and  training,  and  foster  communication 
throughout  the  community — much  has  been  accomplished 

•  OSD  cannot  do  everything... nor  should  we 

•  Challenges  Remain 

-  Getting  ALL  programs  properly  structured  through  effective 
planning — SEP/TEMP/Risk  Management  Plan/Exit  Criteria/ASR 

-  Refocusing  Acquirer  and  Supplier  on  technical  management  of 
programs 

-  Ensuring  adequate  Government  technical  resources 


Services  and  Agencies,  along  with  Industry,  must 
take  ownership  of  the  institutionalization  of  SE 
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Questions. ..perhaps  Answers 
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What  should  be  addressed 
in  a  sound  technical  plan 
for  a  program? 
(Continued) 


Technical  Planning  Area  1 


•  Program  Requirements 

-Capabilities,  CONOPS,  KPPs 

-  Statutory/regulatory 

-  Specified/derived  performance 

-  Certifications 

-  Design  considerations 
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Capabilities,  CONOPS,  KPPs 


•  Most  programs  have  KPPs,  then  what? 

•  What  is  the  plan  for  how  they  will  be  managed?, 
Tested-to,  traded  against  other  requirements? 

•  How  are  they  captured,  analyzed,  decomposed,  and 
allocated? 

•  Who  is  responsible  for  the  above? 

•  Who  are  the  stakeholders? 
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Statutory/Regulatory 


•  What  are  all  of  the  statutory  and  regulatory  requirements? 

•  What  is  the  plan  for  capturing  and  managing  this  set  of 
requirements? 

•  Beyond  the  statutes  and  regulations  themselves,  how  are  the 
specific  requirements  identified,  analyzed,  decomposed,  and 
allocated? 

•  How  are  these  requirements  to  be  managed  in  an  integrated 
framework  with  KPPs,  etc. 

•  Who  is  responsible  for  the  above? 

•  Who  are  the  stakeholders? 
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Specified/Derived  Performance 


•  What  is  the  plan  for  managing  and  integrating  the  totality  of 
specified  performance  (per  the  applicable  system  spec  or 
performance  spec)? 

•  Who  is  responsible  for  derivation,  decomposition,  and 
allocation  of  requirements? 

•  What  tools  will  be  used  and  what  organizational  elements  are 
responsible  for  ensuring  requirements  traceability? 

•  How  will  the  program  ensure  that  these  requirements  are 
managed  across  contractual  boundaries  (subsystem 
suppliers)? 
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Certifications 


•  What  are  all  of  the  certifications  to  which  the  program  is 
subject?  How  does  the  program  ensure  that  all  applicable 
certification  are  identified? 

•  Who  are  the  stakeholder  organizations  responsible  for 
certification  requirements? 

•  How  will  the  program  ensure  that  all  of  the  certification 
requirements  find  their  way  into  the  integrated  set  of 
requirements? 

•  How  are  the  respective  certification  processes  integrated  with 
the  program’s  own  design,  development,  and  test  approach? 
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Design  Considerations 


•  What  is  the  program’s  approach  to  addressing  and 
managing  the  ever-growing  list  of  potentially 
applicable  design  considerations? 

•  How  are  these  requirements  integrated  with  the 
other  requirements  (both  specified  and  derived)? 

•  Who  is  responsible  for  addressing  these 
requirements  that  span  a  broad  range  of  domains 
and  subject  matter  areas? 

•  How  will  technical  budgets  be  established,  allocated, 
and  managed  (reliability,  weight,  etc.)? 
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Technical  Planning  Area  2 


•  Technical  Staffing/Organization 
-Technical  authority 
-  Lead  Systems  Engineer 
-IPT  organization 
-IPT  coordination 
-Organizational  depth 
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Technical  Authority 


•  What  technical  authority  (functional  leads)  is  implied  from  the 
integrated  set  of  Requirements  (KPPs,  statutory,  regulatory, 
specified,  certification,  design  considerations)? 

•  What  organizations  will  be  supporting  the  program  in  the  role 
of  technical  authority? 

•  How  will  the  program  leverage  tech  authority  resources  and 
balance  the  need  to  support  with  the  budgetary  constraints? 

•  What  is  the  program’s  approach  to  integrating  technical 
authority  on  the  appropriate  IPTs? 
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Lead  Systems  Engineer 


•  Does  the  program  have  a  lead  systems  engineer? 
Who  is  this  person  and  how  do  they  interact  with  SE 
technical  authority? 

•  What  is  the  LSE’s  role  and  authority  on  the  program 
relative  to  SE  processes  and  products  (technical 
reviews,  technical  baselines,  etc)? 

•  How  will  the  LSE  and  the  PM  coordinate  in  technical 
management? 

•  What  is  the  plan  for  how  the  LSE  will  manage  SE 
activities  vertically  and  horizontally  across  IPTs? 
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IPT  Organization 


•  What  is  the  program’s  approach  to  the  IPT 
structure? 

•  How  does  the  IPT  structure  relate  to  the  program’s 
products? 

•  What  is  the  program’s  plan  for  alignment  of  the  WBS 
with  the  IPT  structure? 

•  How  are  the  IPTs  populated  to  integrate  all 
stakeholders  (users,  developers,  testers,  technical 
authority,  and  design  considerations) 
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IPT  Coordination 


•  How  are  the  systems  engineering  processes 
managed  and  controlled  across  the  IPTs? 

•  How  are  the  systems  engineering  products 
(requirements  and  technical  baselines)  managed 
and  controlled  across  the  IPTs? 

•  If  there  are  functional  as  well  as  product  IPTs,  what 
is  the  program’s  approach  to  the  respective  roles? 

•  How  does  the  program’s  IPT  structure  and  operation 
provide  for  system  integration? 
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Organizational  Depth 


•  Does  the  SEP  address  overall  organization  of 
Government  and  contractor  systems  engineering 
tasks,  activities,  and  responsibilities  (requirements, 
technical  baseline,  technical  reviews,  etc)  from  prime 
contractor  down  to  lowest  level  supplier? 

•  If  a  part  of  system-of-systems  or  family-of-systems, 
what  is  the  interaction  with  higher  and  peer 
organizations  and  authorities  regarding  design 
trades? 
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Technical  Planning  Area  3 


•  Technical  Baseline  Management 

-Who  is  responsible 

-  Definition  of  baselines 

-  Requirements  traceability 
-Specification  tree  and  WBS  link 
-Technical  maturity 
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Who  is  Responsible? 


•  What  is  the  program’s  approach  to  overall 
management  of  the  technical  baselines?  Who  are 
the  participants  across  the  program? 

•  How  does  this  approach  relate  to  the  roles  and 
responsibilities  of  the  lead  systems  engineer,  the  IPT 
leads,  and  any  functional  IPTs  assigned  for 
requirements  management? 

•  What  is  the  plan  for  technical  baseline  management 
across  IPTs  and  across  the  program  office,  prime, 
and  sub-suppliers? 
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Definition  of  Baselines 


•  What  is  the  program’s  approach  to  utilization  of 
technical  baselines  as  a  technical  management  tool? 

•  How  are  technical  baselines  used  to  across  the 
domains  of  functional,  allocated,  and  product 
attributes? 

•  What  is  the  program’s  approach  to  these  baselines 
relative  to  the  WBS?  EVM?  TPMs? 

•  How  is  the  program  using  technical  reviews  to 
manage  the  technical  baselines  and  assess 
technical  maturity  and  risk? 
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Requirements  Traceability 


•  How  are  requirements  (KPP,  statutory,  regulatory,  specified, 
derived,  certification,  design  considerations,  etc)  tracked  from 
source  to  (and  throughout)  the  program  technical  baseline 
and  specification  tree? 

-  What  is  the  program’s  plan  to  ensure  that  there  are  no  “orphan”  or 
“childless”  requirements? 

-  What  tools  will  be  used  an  by  whom? 

•  Is  this  traceability  addressed  in  the  requirements 
management  and  configuration  management  planning? 

•  Does  the  traceability  extend  to  the  verification  and  validation 
requirements  and  planning? 
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Specification  Tree  and  WBS  Link 


•  What  is  the  program’s  WBS  and  how  does  it  relate  to  the  end 
item  configuration? 

-  Is  the  plan  reflective  of  an  understanding  as  to  how  many  CIs  are 
planned? 

-  What  is  the  program’s  approach  to  technical  baseline  specifications 
(system  spec(s),  functional  spec(s),  subsystem  specs(s),  design 
spec(s)  and  is  there  alignment  between  the  WBS  and  the  planned 
technical  baselines  (specs)? 

•  What  is  the  program’s  approach  to  managing  against  the 
WBS  across  contractual  boundaries? 

•  How  does  the  program  plan  to  use  the  WBS  and  the 
specifications  as  a  technical  management  tool  across  the  SE 
tasks? 

•  What  is  the  program’s  approach  down  to  the  Cl  level? 
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Technical  Maturity 


•  What  is  the  program’s  plan  to  measure  technical 
maturity  (as  opposed  to  TPM  tracking)? 

•  How  is  the  program  using  the  SE  products  of  the 
technical  baseline  (functional,  allocated,  and 
product)  to  gauge  technical  maturity? 

•  Who  is  involved  in  this  assessment  from  a 
stakeholder  perspective? 

•  Has  the  program  established  maturity  criteria  and 
what  is  the  plan  for  application  of  these  criteria 
across  the  WBS  and  down  to  the  Cl  level? 
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Technical  Planning  Area  4 


•  Technical  Review  Planning 

-  Event-driven  reviews 

-  Management  of  reviews 
-Technical  authority  chair 

-  Key  stakeholder  participation 

-  Peer  participation 
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Event-driven  Reviews 


•  What  is  the  program’s  approach  to  executing  event-driven 
reviews? 

-  Are  best  practice  criteria  being  applied? 

-  Is  technical  authority  being  engaged  to  develop  criteria  for  specific 
reviews  and  who  will  assess  readiness  for  the  conduct  of  the  review? 

-  How  many? 

-  For  what? 

•  Is  the  timing  of  the  reviews  in  program  plan  reflective  of  the 
achievable  technical  maturity  required  (per  best  practice)  for 
the  review? 

•  What  is  the  program’s  plan  for  ensuring  that  reviews  are 
event  vice  schedule  driven? 


51 


Management  of  Reviews 


•  What  is  the  program’s  approach  for  oversight  and 
conduct  of  all  technical  reviews? 

•  Who  is  responsible? 

•  How  is  technical  authority  involved  /  engaged? 

•  How  is  the  program  planning  to  ensure  that  technical 
products  subject  to  the  review  are  available  prior  and 
to  the  appropriate  stakeholders? 

•  What  is  the  plan  for  integrating  the  outcomes  of  the 
technical  reviews  into  the  program’s  plan  forward? 
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Technical  Authority  Chair 


•  What  is  the  program’s  approach  to  chairing  of 
technical  reviews? 

•  How  will  the  program  ensure  that  reviews  are 
conducted  to  “best  practice”? 

•  How  is  technical  authority  to  be  engaged  and  what  is 
the  approach  for  system-level  and  subsystem-level 
reviews? 

•  How  will  the  program  manager,  lead  systems 
engineer,  and  technical  authority  collaborate? 


53 


Key  Stakeholder  Participation 


•  What  is  the  program’s  approach  to  attendance  at  reviews? 

•  What  is  the  plan  to  ensure  that  stakeholders  are  involved  at 
key  decision  points?  Example:  airworthiness  certifiers  at 
technical  reviews) 

•  Are  Users,  Testers,  and  Logisticians  involved  in  the  execution 
of  systems  engineering? 

•  Are  representative  offices  from  “design  considerations”  areas 
involved? 

•  How  is  the  program  office  reconciling  resource  realities  with 
these  technical  needs? 
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Peer  Participation 


•  What  is  the  program’s  approach  to  addressing  “peer”  of  third 
party  insight  to  the  program? 

•  Who  are  these  peers  and  from  where  will  they  be  attained? 

•  Are  there  provisions  for  cross-talk  at  the  peer  level  at  key 
points  such  as  the  technical  reviews? 

•  What  is  the  program’s  approach  to  the  areas  (subject  matter 
areas)  that  peer  insight  is  most  critical? 

•  Peer  participation  at  the  SE  leadership  level?  Beyond? 
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Technical  Planning  Area  5 


•  Integration  with  Overall  Management  of  the  Program 

-Linkage  with  other  program  plans 

-  Program  manager’s  role  in  technical  reviews 

-  Risk  management  integration 
-Test  and  logistics  integration 
-Contracting  considerations 
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Linkage  with  Other  Program  Plans 


•  What  is  the  program’s  approach  to  linking  and  integrating  SE 
(technical  management)  with  other  management  efforts? 

•  Was  the  SEP  the  basis  for  the  IMP/IMS? 

•  Was  the  IMS  the  basis  for  the  IBR/cost  account/EVM 
approach? 

•  Were  the  technical  baselines  (across  the  WBS)  incorporated 
as  products  in  EVM? 

•  Are  the  technical  review  risk  assessments  treated  as  inputs  to 
the  risk  management  approach? 

•  Is  the  independent  cost  estimate  based  on  systems 
engineering? 

•  Does  the  PM’s  program  management  plan  use  SE  as  the 
technical  management  arm? 
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Program  Manager’s  Role  in  Reviews 


•  Is  the  SEP  indicative  of  the  PM  using  the  technical 
reviews  as  a  technical  product  to  him/her? 

•  Is  the  PM  (acquirer  and  supplier)  to  be  an  active 
participant  on  the  technical  review  board? 

•  Is  the  program  manager  planning  to  self-chair 
technical  reviews? 
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Risk  Management  Integration 


•  How  are  risk  and  systems  engineering  linked  in  the 
program  planning? 

•  Does  the  SEP  reflect  strong  linkages  between  the 
technical  reviews  and  the  program’s  risk  assessment 
process  (i.e.  risks  of  successful  completion  of  the 
next  technical  review)? 

•  Does  the  plan  reflect  the  decision-making  process 
necessary  to  mitigate  risks? 

•  Does  the  risk  management  plan  refer  to  the  SEP  at 
an  operational  planning  level? 
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Test  and  Logistics  Integration 


•  What  is  the  program’s  approach  to  integration  of  the 
T&E  communities  in  the  SE  process? 

•  Are  the  verification  and  validation  plans  part  of  the 
technical  baselines? 

•  Is  supportability  and  the  support  systems  part  of  the 
technical  baseline? 

•  Are  the  testers  and  logisticians  involved  in  the 
technical  reviews? 

•  Does  the  TEMP  and  ILSP  align  with  the  SEP? 
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Contracting  Considerations 


•  Are  there  provisions  in  the  contract  to  incentivize  best 
systems  engineering  practices  as  applied  on  the  program? 

•  Are  technical  reviews  used  as  a  basis  for  progress 
payments?  (BAD) 

•  Are  there  provisions  in  the  prime  and  sub-supplier  contracts 
for  the  execution  of  technical  management  across  contractual 
boundaries  (SE  processes  and  products  extend  across  the 
team)? 

•  Is  systems  engineering  treated  in  the  contract  as  an  integral 
part  of  the  development  or  as  an  overhead  function  with  no 
product? 

•  Has  the  supplier’s  systems  engineering  approach  been 
“piecemealed”  or  “edited”  to  remove  seemingly  non-value- 
added  work? 
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